Software developers create software programs by writing source code using one or more programming languages. The programs that developers use to write the source code are called software development tools. These development tools typically provide various debugging features that allow software developers to inspect how a particular program operates, and to step through the particular program and identify possible defects. When a developer changes a particular program, the change often has unintended consequences and breaks some functionality in another area. The developer can use the manual approach of stepping through the program using a debugger to try and track down the problem by comparing the observed steps to his or her expectations. However, there is also a more automatic approach: If a trace is available from a previous test run that passed the test, and from a run that did not pass the test, then a tool can compare the two traces. If the traced functionality is deterministic, the first difference usually points to the place in the code (or at least close to it) where the change broke the functionality. This method is called differential debugging.
Unfortunately, the differential debugging method does not work when multiple threads or several processes or machines are involved. The traces from different threads/processes interleave in a random way that changes every time, so it is difficult to figure out which difference is due to the different timing and what is the first “real” difference in the functionality.
Also, the testing of concurrent programs is hard in general since the execution is not deterministic, so bugs may occur only under some rare conditions and are hard to reproduce. There are currently two different approaches to this problem. First, a test case can be run a certain number of times. The problem with this approach is that it tends to miss problems that occur with a low probability. A second approach to testing concurrent programs is model checking. Model checking is where the program is modeled and all possible executions are investigated. The problem with model checking is that it is typically only feasible for small programs since the number of possible executions grows exponentially with the number of choices.